Using the Experience Sampling Method to Evaluate Ubicomp Applications

T he evaluation process is critical for successfully deploying new technologies. You can use evaluations throughout a technology's development to inspire new applications by identifying unmet user needs and to help learn whether your applications meet the needs of your users. If your applications are not meeting your users' needs, evaluations can help identify the reasons. There is a class of ubicomp devices designed to accompany users through different tasks and in changing contexts and environments. Designing for this class is particularly challenging, especially given the variability of users, uses, and environments involved. For example, a mobile phone user is likely to use the device at the office, at lunch, in the car, at the store, and at home. Such settings cannot be reasonably approximated in a traditional laboratory. Recently, the Intel Research Personal Server team wanted help creating realistic usage scenarios. The Personal Server is a mobile device that lets the user store and access data and applications through interfaces found in the environment. The Personal Server does not use a display; it wirelessly connects to nearby input and output devices. 1 To help create the usage scenarios, the Personal Server team wanted to gain a better understanding of people's information needs. Our challenge as evaluators was to find a technique that would help the Personal Server team collect appropriate data. Given that the Personal Server application has a broad group of target users, we needed an evaluation technique that would accommodate many participants in several environments. People will adopt and use ubicomp applications— including the Personal Server—in several settings and for potentially different tasks, so appropriate evaluation techniques must take place in those settings and explore those different tasks. Such requirements are often called in situ (in the actual situation) or ecologically valid (" the occurrence and distribution of stimulus variables in the natural or customary habitat of an individual "). 2 The evaluation techniques used for these ubicomp applications would ideally take place in situ, involve several participants, take place over time, and collect both qualitative and quantitative data. To help with the Personal Server evaluation, we used a technique from the field of psychology—called the Experience Sampling Method 3 —that other researchers have found to be effective for learning about situations and person-situation interactions. The technique compares most closely with recall-based self-reporting techniques such as interviews, traditional surveys, and diaries. In other contexts, researchers have called …